Skip to content

ci: seed preview database locally then bulk-import to Turso#1144

Draft
busbyk wants to merge 1 commit into
mainfrom
busbyk/preview-seed-local-import
Draft

ci: seed preview database locally then bulk-import to Turso#1144
busbyk wants to merge 1 commit into
mainfrom
busbyk/preview-seed-local-import

Conversation

@busbyk

@busbyk busbyk commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Description

The preview deploy seeded each PR's fresh Turso (libSQL) database by running pnpm seed:standalone with DATABASE_URI pointed at remote Turso. Payload's seed performs thousands of individual create() calls, and against a remote database each one is a network round-trip — so the step is latency-bound (~289s on ubuntu-latest, and long enough on higher-latency runners to lose the connection mid-seed).

This seeds into a local SQLite file instead, then bulk-imports that file into the fresh Turso preview DB in a single operation. The end state is identical — a fully seeded Turso preview DB — but the seed/import drops from ~5 min to seconds and no longer depends on sustained low-latency round-trips, so it's runner-agnostic.

Related Issues

Refs #1143

Key Changes

  • Seed to a local file, in push mode. The seed step now targets file:./preview-seed.db and overrides NODE_ENV off production so the SQLite adapter runs in push mode and builds the current schema directly from the Payload config — the same fast path the CI build job uses.
    • Running migrations from zero isn't viable on this repo: the 20260505_..._backfill_document_references data migration reads collections via the Local API, whose query is built from the current config and references block tables (e.g. pages_blocks_form_embed) that a later migration adds — so on a fresh DB the query hits no such table and aborts. The old remote-Turso path would hit this too.
  • Bulk-import with turso db create --from-dump (a sqlite3 .dump), reordered to run after the local seed. --from-dump is used rather than --from-file/db import, which require WAL mode and mis-parse valid WAL databases on sqlite3 ≥ 3.50 (tursodatabase/turso-cli#1030); a plain SQL dump is runner-agnostic.
  • Removed the two remote-Turso steps (Run database migrations + Seed the database). Vercel env wiring, build, deploy, and aliasing are unchanged — the deployed preview still points at the populated Turso DB.
  • Kept runs-on: ubuntu-latest; moving preview onto a Blacksmith runner is the trivial follow-up called out as out-of-scope in Speed up preview seeding: seed locally, then bulk-import to Turso #1143.

How to test

The preview workflow runs on this PR (opened/synchronize), so its own preview deployment is the end-to-end check: after the Preview environment is approved, confirm https://<ref>.preview.avy-fx.org comes up fully seeded (tenants, users, pages) and that the Seed a local SQLite database + Create the preview database from the seeded file steps complete in well under a minute.

Local verification performed while developing:

  • Push-mode seed of a fresh local file → 4 tenants / 19 users / 112 pages.
  • sqlite3 .dump (~1.5 MB) reloaded into a fresh DB preserves all 248 tables and identical row counts — a faithful proxy for Turso's --from-dump, which replays the same SQL. (A live Turso round-trip wasn't run locally; the PR's preview run exercises it for real.)

Screenshots / Demo video

n/a

Migration Explanation

No schema/migration changes — workflow-only.

Future enhancements / Questions

🤖 Generated with Claude Code


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

The preview deploy seeded each PR's fresh Turso DB by running
seed:standalone with DATABASE_URI pointed at remote Turso. Payload's
seed does thousands of per-row create() calls, so against a remote
libSQL database every row is a network round-trip (~289s on
ubuntu-latest, and long enough on higher-latency runners to lose the
connection mid-seed).

Seed into a local SQLite file instead, then bulk-import that file into
the fresh Turso preview DB in a single operation:

- Seed step overrides NODE_ENV off `production` so the SQLite adapter
  runs in push mode and builds the current schema directly from the
  Payload config — the same fast path the CI build job uses. Running
  migrations from zero isn't viable here: the backfill_document_references
  data migration reads collections whose block tables a later migration
  adds, so the query hits `no such table` on a fresh DB.
- Import with `turso db create --from-dump` (a sqlite3 .dump) rather than
  `--from-file`, which requires WAL mode and mis-parses valid WAL
  databases on sqlite3 >= 3.50 (tursodatabase/turso-cli#1030).

The end state is identical — a fully seeded Turso preview DB — but the
seed/import drops from ~5 min to seconds and no longer depends on
sustained low-latency round-trips, so it works on any runner.

Refs #1143

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

@busbyk busbyk marked this pull request as draft July 9, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant